Data corruption is a prevalent problem in storage systems. While there are many forms of data corruption, there are also many ways to prevent data corruption, each having their own advantages and disadvantages. For example, enterprise class servers use error checking and correcting caches and memory to protect against single and double bit errors. System buses have similar protective measures such as parity. Communications over a network are typically protected by checksums. Many storage systems utilize RAID (redundant array of inexpensive disk) technology to protect against disk failure. In the case of hardware RAID, the array firmware will often use advanced checksumming techniques and media scrubbing to detect and correct potential errors. The disk drives may also feature sophisticated error corrective measures, and storage protocols such as Fibre Channel (FC) and iSCSI (Internet small computer system interface) often feature a cyclic redundancy check (CRC) which guards against data corruption on the wire.
A common trait in most of the existing protective measures is that they work in their own isolated domains or at best between two adjacent nodes in the input/output (I/O) path. In many applications, however, it is desirable (or required) to ensure true end-to-end data integrity (i.e., between a host device and a corresponding destination device). An extension to the SCSI family of protocols allows extra protective measures, including a checksum, to be included in an I/O request. This appended data is referred to as integrity metadata or protection information. Unfortunately, the SCSI protection envelope only covers the path between the I/O controller and the storage device. To remedy this, Oracle and select industry partners collaborated to design a method of exposing the data integrity features to the operating system. This technology, known as data integrity extensions, allows the operating system to generate protection data that will be verified as the request goes through the entire I/O stack, thereby providing a degree of end-to-end data integrity.
T10 is a technical committee of the International Committee for Information Technology Standards (INCITS) standards body responsible for the SCSI family of protocols. Data corruption has been a known problem in the storage industry and T10 has provided a means to prevent such data corruption by extending the SCSI protocol to allow integrity metadata to be included in an I/O request. The extension to the SCSI block device protocol is called the data integrity field (DIF). In a DIF implementation, a storage device having a standard hardware sector size of 512 bytes is reformatted to 520- or 528-byte sectors; the extra eight or sixteen bytes in each sector, respectively, are used internally by the array firmware for integrity checks. The main difference between DIF and proprietary array firmware is that the format of the extra eight or sixteen bytes of information per sector is well-defined as part of an open standard. This means that every node in the I/O path can participate in generating and verifying the integrity metadata.
Unfortunately, however, the existing T10 DIF mechanism requires significant overhead, particularly for data retransmission. Furthermore, systems implementing a conventional T10 DIF standard suffer performance degradation, including, among other disadvantages, reduced data transfer rates.